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FOREWORD 


The  U.S.  Army  Research  Institute  for  the  Behavioral  and  Social  Sciences  (ARI),  as  part 
of  its  WEBTRAIN  project,  is  investigating  the  use  of  distributed-learning  technology  for  use  by 
soldiers.  ARI  seeks  to  provide  guidance  to  the  Army  as  it  transforms  from  classroom-based 
instruction  to  more  distributed,  soldier-centric,  and  collaborative  instruction  methodologies. 

With  the  U.S.  Army  transforming  many  of  its  courses  to  a  distributed-learning  format, 
new  methodologies  are  needed  to  take  full  advantage  of  current  technology.  In  this  report,  the 
distributed-learning  portion  of  the  Armor  Captains’  Career  Course  for  Reserve  Component 
Officers  was  examined.  Findings  of  this  report  are  relevant  to  course  designers  interested  in 
implementing  collaborative  tools  for  distributed  learning.  This  research  was  briefed  to  Assistant 
Deputy  Chief  of  Staff  -Training,  TRADOC  on  17  October,  2002.  Some  portions  of  the  research 
were  published  in  journal  form  (Orvis,  Wisher,  Bonk  &  Olson,  2002)  and  as  an  ARI  Research 
Note  (Bonk,  Olson,  Wisher,  &  Orvis,  2002).  The  results  and  recommendations  of  this  research 
were  also  presented  at  the  Army  Science  Conference  on  3  December  2002. 


MICHAEL  G.  RUMSEY 


Acting  Technical  Director 
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WEB-BASED  COLLABORATIVE  LEARNING:  COMMUNICATION  BETWEEN 
LEARNERS  WITHIN  A  VIRTUAL  TACTICAL  OPERATIONS  CENTER 


EXECUTIVE  SUMMARY 


Research  Requirement: 

The  U.S.  Army  Armor  School  (USAARMS)  redesigned  the  Armor  Officers’  Advanced 
Course  for  Reserve  Component  officers  (AO AC  RC)  so  that  the  first  phase  could  be  delivered 
via  web-based  instruction  instead  of  via  the  previous  method,  written  correspondence.  The 
Director  of  the  USAARMS  requested  that  the  U.S.  Army  Research  Institute’s  Advanced 
Training  Methods  Research  Unit  (ARI-ATMRU)  evaluate  the  distributed  learning  phase. 

Procedure: 

During  synchronous  collaborative  instruction,  students  used  a  computer-based  text 
messaging  function  to  communicate  with  other  students  in  a  Virtual  Tactical  Operations  Center 
(VTOC).  The  content  of  the  text  messages  between  students  was  analyzed  across  six  training 
sessions.  Additionally,  to  determine  strengths  and  weaknesses  of  the  distributed  learning  portion 
of  the  course,  interviews  were  conducted  with  two  groups  of  four  students,  three  instructors,  and 
the  course  supervisor. 

Findings:  ■ 

Three  types  of  text  communication  were  identified:  task-related,  social,  or  technology- 
related.  Task-related  messages  involved  statements  oriented  toward  the  course  exercise.  Social 
interactions  were  messages  not  related  to  the  task,  but  involved  personal  content.  Technology- 
related  interactions  involved  the  equipment  used  for  the  exercise.  The  relative  frequency  of  the 
different  types  of  text  messaging  changed  across  sessions.  During  the  initial  session,  moderate 
levels  of  on-task,  social,  and  technology-related  communication  occurred.  Over  the  next  few 
sessions,  the  level  of  social  communication  dropped,  but  rebounded  during  the  final  session, 
while  the  frequency  of  technology-related  communication  dropped  and  remained  low.  The 
relative  frequency  of  on-task  communication  peaked  during  the  middle  sessions,  but  was  usually 
the  most  frequent  type  of  communication  across  sessions. 

The  interviews  with  students,  course  instructors,  and  the  course  supervisor  revealed 
additional  information.  For  example,  during  the  synchronous  distributed  learning  phase,  the  role 
of  the  instructor  was  seen  as  a  facilitator  instead  of  as  a  lecturer.  Also,  to  limit  the  frequency  of 
problems  that  may  interfere  with  instruction,  the  course  supervisor  recommended  that  course 
designers  should  not  stretch  the  limits  of  the  instructional  technology. 

Utilization  of  Findings: 

The  change  in  relative  frequencies  in  the  nature  of  text  messaging  provides  insight  to  the 
development  of  a  virtual  training  team,  and  should  be  useful  to  both  course  developers  and 
instructors  interested  in  such  social  interaction  and  problem  solving.  In  addition,  the  findings 
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from  the  participant  interviews  regarding  the  facilitative  role  played  by  instructors  should  be 
particularly  useful  to  instructors  of  distributed  courses. 
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Web-based  Collaborative  Learning: 

Communication  between  Learners  within  a  Virtual  Tactical  Operations  Center 

INTRODUCTION 

The  U.S.  Army  is  embracing  distributed  learning  (DL)  as  an  agent  of  change  in  the  areas 
of  training  and  education.  The  Army  is  in  the  process  of  developing  more  than  525  DL  courses 
by  2010.  Hundreds  of  digital  training  facilities  are  currently  available  in  both  the  regular  Army 
and  reserve  components  to  provide  access  to  technology  and  learning  content  (Program 
Management  Office,  2000).  As  of  April  2003,  the  Army  University  Access  Online  program,  also 
known  as  eArmyU,  has  enrolled  35,000  soldiers,  equipped  them  with  laptops  and  Internet  access 
accounts,  and  encouraged  them  to  earn  a  college  degree  while  serving  on  active  duty.  In  the 
National  Defense  Authorization  Act  of  2002,  Congress  passed  legislation  allowing  reservists  to 
be  compensated  for  completion  of  asynchronous  DL  courses.  Clearly,  the  interest  in  DL  from  an 
infrastructure  and  policy  perspective  is  growing,  although  questions  surrounding  its  optimal  use 
remain. 


While  delivery  of  DL  over  the  Web  is  still  relatively  new,  the  benefits  have  already  been 
demonstrated  in  many  ways,  such  as  decreased  cost  and  increased  time  management  (Army 
Logistician,  2001;  Sanders  &  Burnside,  2001;  Deloughry,  1995;  Howard,  1992;  Kulik,  &  Kulik, 
1991;  Metzko,  Redding  &  Fletcher,  1996).  Metzke,  Redding,  &  Fletcher  (1996)  reviewed  30 
studies  on  DL  instruction  and  found  that  students  needed  a  median  of  30  percent  less  time  to 
complete  distributed  courses  as  compared  to  traditional  instruction.  In  addition,  students  who 
completed  courses  via  DL  attained  a  higher  level  of  achievement  when  compared  to  traditional 
instruction,  an  average  increase  of  nearly  0.5  standard  deviations. 

While  DL  has  its  advantages,  there  are  some  areas  where  it  still  lags  behind  face-to-face 
instruction,  such  as  ease  of  communication  and  the  development  of  deeper  relationships  at  a 
faster  pace  (Olson  &  Olson,  2000).  In  the  current  research,  we  analyzed  the  use  of  text 
communication  for  Web-based  collaboration  by  students  during  synchronous  DL. 

Web-based  Collaborative  Learning 

Collaboration  is  a  key  part  of  the  “Learner-Centered  Psychological  Principles”  promoted 
by  the  American  Psychological  Association  (APA,  1997).  Through  collaborative  tasks,  such  as 
discussing,  summarizing,  clarifying,  and  integrating  course  content  into  an  overall  framework, 
learners  acquire  knowledge  and  gain  a  deeper  understanding  (Deatz  &  Campbell,  2001; 
Palincsar,  1998).  Collaboration  among  learners  allows  for  the  synergistic  building  of 
knowledge.  Numerous  studies  have  shown  that  the  use  of  collaborative  learning  exercises  leads 
to  improved  knowledge  acquisition  as  compared  to  traditional  classroom  instruction  without 
collaborative  exercises  (Alkhateeb  &  Jumaa,  2002;  Bonk  &  Wisher,  2000;  Fisher  &  Coleman, 
2001-2002;  Kang,  1998). 

The  use  of  collaborative  tools  during  on-line  exercises  can  be  helpful,  not  only  for 
completing  the  task  at  hand,  but  also  for  social  functions  such  as  team  building  and  group 
cohesion  (Bonk  &  Wisher,  2000;  Kang,  1998).  People  in  DL  courses  sometimes  feel  isolated 
because  there  is  a  lack  of  natural  interaction  with  class  members  as  compared  to  traditional 
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classrooms  where  learners  have  the  opportunity  to  meet  one  another.  Collaborative  tools  can  help 
to  reduce  these  feelings  of  isolation  experienced  by  distributed  learners  (Muilenburg  &  Berge, 
2001). 

Two  of  the  more  frequent  Web-based  tools  used  for  collaboration  are  text  messaging 
(e.g.,  instant  messaging  or  text  chat)  and  e-mail.  Text  messaging  allows  one  learner  to  transmit  a 
string  of  text  that  immediately  is  presented  to  another  online  learner.  It  is  considered 
synchronous  because  there  is  essentially  no  delay  in  information  transmission  between  the  sender 
and  the  receiver.  E-mail,  however,  is  considered  asynchronous  because  there  is  typically  a  delay 
in  the  receipt  of  the  message,  since  only  one  participant  might  be  online  at  a  given  time.  When 
used  properly,  both  e-mail  and  text  messaging  are  useful  in  DL  courses  (Bonk  &  Wisher,  2000; 
Kang,  1998). 

Asynchronous  text  communication  has  advantages,  such  as  allowing  for  more  time 
flexibility.  However,  because  of  the  immediacy  of  the  communication  users  feel  more  connected 
with  synchronous  communication.  Pena-Shaff,  Martin,  &  Gay  (2001)  found  that  synchronous 
text  messaging  provided  for  more  task  collaboration  than  communication  via  asynchronous 
methods.  In  addition,  participants  also  showed  increased  socializing  using  text  messaging  in 
comparison  to  asynchronous  communication.  Hudson  and  Bruckman  (2002)  concluded  that  the 
“realtime”  aspect  of  text  messaging  is  a  key  factor  leading  to  effective  social  interactions.  These 
studies  suggest  that  when  users  simultaneously  work  together,  or  collaborate,  text  messaging  is 
an  effective  communication  mode. 

In  the  course  examined  in  this  report,  a  text  messaging  system  was  used  during 
synchronous  online  training,  allowing  the  participants  to  communicate  spontaneously.  In  the 
current  research,  we  assessed  the  use  of  text  messaging  and  the  perspectives  of  students, 
instructors  and  the  course  administrator  through  the  use  of  group  interviews. 

Armor  Captains’  Career  Course  Background 

The  purpose  of  the  Armor  Captains  Career  Course  (AC3)  is  to  train  captains  both  for 
command  positions  and  positions  as  assistant  operations  officers  in  a  tactical  operations  center 
(Sanders  &  Burnside,  2001).  In  an  earlier  version  of  the  course  for  reservists,  the  first  of  two 
phases  was  presented  via  written  correspondence  before  a  two-week  in-residence  training  phase. 
This  training  schedule  was  required  because  training  in  the  reserves  is  limited  to  39  days  per 
year,  generally  as  one  weekend  per  month  and  one  two-week  period.  The  course  requirements, 
therefore,  had  to  be  tailored  for  such  an  intermittent  training  schedule.  The  use  of  DL,  with  its 
ability  to  offer  training  anytime  and  anywhere,  offers  obvious  flexibility  within  such  a  restricted 
training  schedule. 

The  current  version  of  the  AC3-DL  course  uses  a  blend  of  two  instructional  phases.  The 
first  phase  was  divided  into  one  asynchronous  Internet-based  set  of  lessons  (Phase  la),  followed 
by  a  series  of  synchronous,  collaborative,  Internet-based  exercises  during  weekend  training 
sessions  (Phase  lb).  The  final  phase  (Phase  2)  consisted  of  face-to-face  resident  training  for 
two-weeks.  According  to  Sanders  and  Burnside  (2001),  the  students  who  participated  in  early 
iterations  of  the  AC3-DL  course  finished  the  DL  lessons  in  significantly  less  time  than  students 
who  completed  the  traditional  correspondence  version.  On  average,  students  who  attended  the 


2 


AC3-DL  course  also  achieved  slightly  higher  scores  on  a  knowledge  test  than  students  who 
completed  the  traditional  classroom-based  course  (Sanders  &  Burnside,  2001).  This  finding 
approached  statistical  significance  (p=.055),  suggesting  that  the  conversion  of  the  course  from  a 
traditional  classroom-based  course  to  a  blended  learning  approach  yielded  better  learning. 

The  initial  phase  of  the  AC3-DL  (asynchronous  lessons)  consisted  of  lessons  provided 
over  the  Web  followed  by  end-of-lesson  tests.  The  soldiers  could  access  course  material  and  take 
the  tests  at  any  time  from  any  computer  with  Internet  access.  During  this  phase,  students  learned 
basic  terms  and  concepts  for  use  later  in  the  course.  Students  needed  to  pass  each  multiple-choice 
test  with  a  score  of  at  least  70  percent  correct  in  order  to  proceed  to  the  next  lesson.  If  students 
scored  lower  than  70  percent,  they  were  allowed  to  retake  the  tests  until  successful.  Students 
received  feedback  immediately  from  the  computer  and  via  email  from  the  instructor.  This  self- 
paced  set  of  lessons  was  comparable  to  a  3-credit  college  level  course.  The  soldiers  in  Sanders 
and  Burnside  (2001)  averaged  7.5  months  to  complete  this  phase. 

For  synchronous  exercises  (Phase  lb),  students  “convened”  in  the  virtual  tactical 
operations  center  (VTOC)  for  training  sessions  lasting  between  four  and  eight  hours  on  two 
consecutive  weekend  days.  The  VTOC  was  a  web-based  rendition  of  an  actual  tactical 
operations  center  (see  Figure  1).  This  phase  consisted  of  students  in  separate  locations 
completing  collaborative  exercises.  The  lessons,  scheduled  for  weekend  training  sessions, 
involved  each  student  performing  a  specific  role  in  time-limited  exercises  requiring  the 
participation  of  all  students.  To  collaborate  during  this  phase,  students  used:  a)  audio 
conferencing  where  they  could  speak  to  any  of  the  other  team  members  and  be  heard  by  all  of  the 
team  members;  b)  a  text  messaging  application  where  they  could  type  a  text  message  to  either  all 
members  or  a  subset  of  the  team;  c)  a  virtual  image  of  the  team  so  that  the  function  of  each 
member  could  be  identified;  and  d)  shared  applications.  For  additional  detail  of  VTOC  and  the 
AC3-DL,  see  Sanders  and  Burnside  (2001)  and  Sanders  and  Guyer  (2001). 

In  the  VTOC,  there  were  four  shared  applications:  a  shared  text  program  where  students 
could  write  operation  orders,  warning  orders  and  other  planning  documents;  a  shared  bookshelf 
where  students  could  display  field  manuals;  a  Mapedit  program  where  students  could  display  and 
modify  map  images;  and  a  3-dimensional  virtual  environment  which  allowed  the  students  to 
“walk”  and  symbolically  comment  on  the  terrain  of  the  mission.  With  these  collaborative  tools, 
they  collectively  solved  problems  concerning  military  operations  and  generated  work  products, 
such  as  a  mission  analysis. 

The  final  phase  of  the  course  consisted  of  in-residence  collaborative  exercises,  where 
students  met  face-to-face  for  the  first  time.  In  this  phase,  the  students  engaged  in  real-life  tactical 
operations  exercises,  applying  the  skills  and  competencies  that  they  learned  during  the  DL  phase 
of  the  course.  As  with  previous  phases,  the  instructors  monitored  each  student’s  performance. 
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Figure  1 .  The  Virtual  Tactical  Operation  Center,  as  it  appears  on  the  computer  screen  of  a  user. 

In  the  current  research,  two  different  types  of  data  were  collected:  text  communication 
and  reactions  to  the  DL  phase  of  the  course.  First,  during  the  collaborative  exercises  of  Phase 
lb,  all  of  the  text  messages  among  the  students  were  recorded  and  analyzed.  The  analysis  of  the 
text  messaging  data  revealed  insight  into  the  ongoing  process  of  VTOC  training  team 
development  and  performance.  Second,  midway  through  the  face-to-face  classroom  section 
(Phase  2),  the  researchers  conducted  interviews  with  students,  instructors,  and  the  course  advisor 
to  obtain  reaction  data  regarding  the  DL  components  of  the  course  (Phase  la  and  Phase  lb). 

This  provided  a  qualitative  assessment  of  the  blended  learning  approach  used  during  this  course. 

METHOD 

Participants 

A  total  of  41  students  who  enrolled  in  the  AC3-DL  course  conducted  by  the  U.  S.  Army 
Armor  School  at  Fort  Knox,  Kentucky,  participated  in  this  research.  All  participants  were  male 
reservists  in  the  Army  National  Guard.  The  participants  were  geographically  dispersed  across 
the  United  States  and  had  never  previously  met  face-to-face.  For  the  synchronous  phase,  the 
course  administrator  assigned  participants  to  five  teams  consisting  of  between  seven  and  nine 
members. 
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Following  completion  of  the  two  distributed  learning  phases  and  midway  through  their 
participation  in  the  face-to-face  section  of  the  course  (Phase  2),  the  researchers  interviewed  eight 
students  from  one  of  the  teams.  The  researchers  also  interviewed  two  of  the  course  instructors 
and  the  course  supervisor. 

Procedure 

Text  messages.  During  VTOC  exercises  in  Phase  lb,  groups  were  scheduled  to  meet  for 
12  hours  on  each  of  seven  weekends  at  monthly  intervals.  The  first  session  introduced  the 
technology,  and  the  remaining  six  sessions  were  task-oriented  training  using  the  VTOC.  The 
researchers  recorded  and  analyzed  six  sessions  of  text  messaging  data,  each  session  spanning  two 
days.  Each  session  began  with  30  minutes  devoted  to  a  communications  check,  followed  by 
approximately  12  hours  of  online  training  per  weekend. 

The  definition  of  one  text  message  was  an  act  of  messaging  with  “a  single,  uninterrupted 
verbalization,  typed  in  the  message  box,”  in  accordance  with  Lebie,  Rhoades,  and  McGrath 
(1996).  Based  on  a  variation  of  the  coding  scheme  used  by  Lebie  et  al.,  two  raters  categorized 
each  text  message  as  one  of  three  types:  task-related,  social,  or  technology-related.  Task-related 
messages  involved  statements  related  to  the  VTOC  exercise  and  the  course  topic.  Social 
interactions  were  messages  involving  personal  content  that  were  not  related  to  the  training  task. 
Social  topics  included  hobbies,  holidays,  and  plans  for  getting  together  during  the  in-residence 
phase  at  Fort  Knox.  Technology-related  messages  involved  discussion  of  the  VTOC  equipment 
used  for  the  exercise,  which  included  how  the  interactive  tools  were  used  or  if  team  members 
were  having  problems  with  the  technology.  Table  1  shows  examples  of  each  type  of  text 
message. 

Table  1 


Examples  of  Task,  Social,  and  Technology  Interactions  in  the  Text  Messaging  Data 


Task 

•  “Remember  in  the  BDE  OPORD-the  BDE  CMDR  wants 
this  to  occur  at  about  this  time” 

•  “I  don’t  see  anything  about  obstacles  in  the  CLOSE 
section” 

•  “I  think  obstacles  in  the  Close  section  of  the  CO  A 
statement  is  a  necessary  evil” 

Social 

•  “Kids  are  great  we  made  breakfast  for  Mom  (wife)” 

•  “Did  you  go  out  for  a  run  last  night?” 

•  “Tell  her  I  said  “Happy  Mothers  Day” 

•  “3  miles  in  24  mins  all  hills” 

•  “If  God  had  meant  for  us  to  run,  he  wouldn’t  have  given 
us  tanks” 

Technology 

•  “Cannot  talk  or  hear... will  try  to  reconnect . ” 

•  “Is  anyone  talking  right  now?  I  think  I  dropped  audio” 

•  “Going  to  reboot” 
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Inter-rater  reliability.  Prior  to  final  coding  of  the  text  message,  inter-rater  reliability  was 
assessed.  Approximately  1 5  percent  of  all  messages  were  selected  at  random,  and  each  of  the 
two  raters  separately  coded  1,034  messaging  acts.  Cohen’s  Kappa  of  inter-rater  reliability 
between  these  two  raters  was  deemed  acceptable  at  k=.  90.  Cohen’s  Kappa  is  a  measure  of  the 
degree  to  which  two  raters  concur  when  placing  items  into  nominal  categories.  The  range  of  k  is 
0-1.0,  and  like  other  correlation  measures  higher  values  indicate  greater  inter-rater  reliability. 
Accounting  for  the  proportion  of  agreement  expected  by  chance,  it  is  better  than  using  percent 
agreement  between  two  raters  (Fleiss,  1981). 

Focus  groups.  Midway  through  the  final  phase  of  training,  one  team  of  eight  students 
was  divided  into  two  groups  of  four  to  participate  in  focus  groups.  During  the  focus  groups,  one 
researcher  asked  each  of  the  two  groups  a  set  of  questions  while  a  second  researcher  recorded  the 
students’  responses.  The  questions  covered  the  asynchronous  phase  and  the  use  of  the  VTOC 
during  the  synchronous  portion  of  the  class  (Phase  lb).  The  questions  are  presented  in  Appendix 
A.  All  participants  who  were  interviewed  had  worked  together  in  the  synchronous  phase  of  the 
course  and  in  the  face-to-face  setting  during  Phase  2. 

Researchers  also  interviewed  two  course  instructors  and  the  DL  Education  Advisor.  The 
questions  asked  to  the  instructors  are  presented  in  Appendix  B.  A  structured  set  of  questions  was 
not  prepared  for  the  course  supervisor.  Instead,  a  researcher  asked  the  supervisor  about  her 
opinion  of  the  AC3-DL  course,  specifically,  and  DL  courses  in  general  to  discern  what  advice 
she  would  give  to  potential  developers  of  DL  projects. 

RESULTS  AND  DISCUSSION 

Two  sets  of  data  were  assessed.  The  first  set  was  quantitative  data  generated  from  the 
analysis  of  the  text  messages  during  the  synchronous  VTOC  training  in  Phase  lb.  The  second 
set  was  qualitative  data  from  the  transcripts  of  the  interviews  conducted  during  Phase  2.  These 
two  sets  of  data  apply  to  different  aspects  of  the  research;  therefore,  the  results  from  each  data  set 
below  was  followed  immediately  by  a  discussion  of  the  findings  for  that  particular  data  set. 

Text  Messaging  Data 

A  total  of  6,806  text  messages  were  recorded  across  all  sessions.  Two  raters  coded  6,705 
of  the  6,806  messages  into  one  of  the  three  interaction  categories;  101  of  the  messages  lacked 
sufficient  information  for  coding,  and  therefore  were  not  included  in  the  following  analysis.  Of 
the  6,705  messages,  54  percent  were  rated  as  task,  30  percent  were  rated  as  social,  and  16 
percent  were  rated  as  technology  related  (see  Table  2). 

Percentages  of  the  types  of  messages  were  used  instead  of  frequency  counts  in  order  to 
standardize  the  results  across  the  six  sessions  because  the  number  of  individuals  and  teams 
differed  across  sessions.  Data  for  some  teams  were  not  available  for  all  sessions,  and  a  few  data 
files  were  accidentally  truncated  during  the  recording  process,  resulting  in  some  losses.  For 
example,  for  one  team,  we  may  have  recorded  both  days  of  sessions  one  through  four  and  only 
the  Saturday  data  of  session  five.  For  all  sessions,  the  data  of  at  least  two  teams  were  recorded. 
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Table  2.  The  percentage  of  text  messages  types  during  synchronous  VTOC  training. 


Type  of  Message 

#  of  Messages 

Percentage  of 

Total  Messages 

Task 

3614 

54% 

Social 

30% 

Technology 

1088 

16% 

Total 

6705 

100% 

The  percentages  of  all  three  message  types  changed  over  the  six  sessions  of  training  with 
the  VTOC.  Task-related  messages  ranged  from  slightly  over  40  percent  of  the  messages  during 
the  first  session  to  over  60  percent  during  the  middle  sessions,  and  then  fell  to  about  40  percent 
during  the  last  session.  Complementary  to  this  trend,  social  text  messaging  was  more  frequent 
during  the  first  and  last  training  sessions  than  during  the  middle  sessions.  The  level  of 
technology-related  messages  steadily  decreased  over  sessions,  from  30-40  percent  during  the 
first  three  sessions  to  less  than  5  percent  during  the  last  session.  Different  trends  were  revealed 
for  task,  social,  and  technology  messages,  as  illustrated  in  Figure  2. 


Average  Across  Groups 
Percentage  of  Chat  Instances 


Sessions 


Figure  2.  The  percent  of  text  messaging  types  (task,  social,  and  technology  related)  across  the  six 
Phase  lb  sessions  using  the  VTOC. 

Three  separate  ANOVAs  were  conducted  to  test  for  statistically  significant  differences 
between  the  six  sessions  for  each  individual  category  of  interaction.  For  these  analyses,  the 
percentage  of  messages  within  each  category  (task,  social,  and  technology)  for  each  participant 
was  compared  across  sessions.  There  were  significant  differences  across  sessions  for  task 
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messages  (F=5.41,  p<.05),  social  messages  (F=  14.20,  p<.05),  and  technology  messages  (F=6.54, 
p<.05).  Post  hoc  analyses  (Tukey’s  test  of  honest  significant  difference)  were  conducted  to 
determine  where  the  significant  differences  occurred  across  sessions.  For  task-related  messages, 
there  was  a  significantly  smaller  percentage  of  messages  in  Session  1  compared  to  Sessions  3, 4, 
and  5.  In  addition,  task  messages  during  Session  4  were  more  likely  than  during  Sessions  1,  2, 
and  6.  For  social  messages,  there  was  a  significantly  greater  percentage  during  Session  1 
compared  to  Sessions  2  and  3,  while  a  significantly  higher  percentage  of  social  messages 
occurred  during  Session  6  than  during  any  of  the  other  sessions.  Finally,  for  technology-related 
messages,  Session  2  produced  a  significantly  higher  percentage  than  Sessions  4,  5,  and  6. 
Additionally,  a  lower  percentage  of  technology-related  messages  occurred  during  Session  6  than 
during  sessions  1,2,  and  3.  All  other  comparisons  were  not  significant. 

The  results  document  that  the  use  of  text  messaging  methodology  for  tracking  group 
interaction  is  viable.  During  the  initial  sessions,  almost  equal  time  was  spent  on  the  task,  getting 
to  know  one  another,  and  figuring  out  how  the  equipment  worked.  As  the  team  progressed  into 
the  middle  sessions,  the  soldiers  focused  on  the  task  more  and  needed  less  help  with  the 
equipment.  As  the  training  sessions  concluded,  almost  no  help  was  needed  regarding  the 
equipment.  Also,  during  the  last  session  the  social  interactions  increased  (the  participants  were 
either  saying  good-bye  or  planning  for  meeting  one  another  during  the  next  phase  of  training). 
The  results  provide  insight  into  collaborative  learning  and  how  the  teams  and  communication 
change  over  time. 

Previous  research  indicates  that  when  Web-based  groups  develop,  the  types  of 
interactions  among  members  change  over  time.  In  research  by  Lebie,  Rhoades,  and  McGrath 
(1996),  the  change  was  similar  to  the  pattern  demonstrated  by  participants  in  the  current 
research.  In  both,  the  frequency  of  communication  regarding  how  to  use  the  technology  or  how 
to  complete  the  functions  of  a  particular  task  decreased  over  time  as  the  users  became  more 
familiar  with  the  equipment  or  more  skilled  with  the  technology.  In  addition,  the  frequency  of 
on-task  communication  was  maintained  at  a  high  level  across  all  sessions,  and  the  frequency  of 
social  interaction  started  high  and  decreased  during  the  initial  sessions. 

One  of  the  trends  of  text  messaging  was  the  steady  decrease  in  technology-related 
messages  as  sessions  progressed.  This  most  likely  was  due  to  increased  familiarization  with  the 
equipment  and  decreased  equipment  difficulties  over  sessions.  Technology-related  problems  are 
one  of  the  main  barriers  to  distributed  education  (Muilenburg  &  Berge,  2001;  Lebie,  Rhoades,  & 
McGrath,  1996;  Olson  &  Olson,  2000).  Some  of  the  difficulties  in  the  AC3  course  were  due  to 
some  students  not  being  familiar  with  the  technology.  When  students  are  introduced  to  new 
technology  they  may  have  difficulty  at  first,  however,  previous  experience  using  technology 
should  increase  the  pace  at  which  new  skills  are  acquired  (Schaab  &  Moses,  2001).  It  might  be 
helpful  to  start  the  course  with  technology  related  training  for  those  who  are  not  familiar  with  the 
technology.  Once  these  skills  are  learned,  they  may  then  be  maintained  and  more  easily  transfer 
to  novel  situations  and  new  training  exercises. 

Lebie,  Rhoades,  and  McGrath  (1996)  found  that  text-based  communication  across  a 
distributed  network  followed  similar  patterns  to  interactions  in  face-to-face  groups.  However, 
the  patterns  took  longer  to  develop  in  the  distributed  groups.  This  suggests  that  the  nature  of 
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communication  in  both  distributed  groups  and  face-to-face  groups  may  be  similar,  but  that  social 
relationships  may  develop  more  slowly. 

Interviews  with  Students 

Each  student  participated  for  approximately  70  hours  in  synchronous  online  instruction. 
They  each  also  had  a  personal  computer  available  for  use  and  access  to  the  Internet  at  both  their 
home  and  work  settings.  Overall,  the  students  favorably  evaluated  their  experience  working  in 
groups  online. 

Students  noted  technological  obstacles.  During  the  focus  group  sessions,  students  noted 
several  technological  problems  and  obstacles  related  to  the  AC3-DL  course.  The  map  editor  and 
virtual  messaging  rooms  were  responsible  for  the  majority  of  students’  technology  problems. 
Such  problems  can  frustrate  students,  interfere  with  learning,  and  negatively  impact  course 
completion  rates  (Olson  &  Olson,  2000). 

The  decreased  frequency  of  technology-related  text  messages  across  sessions  suggests 
that  students  overcame  the  difficulties  that  they  experienced  with  the  VTOC.  This  would  be  in 
agreement  with  Lebie,  Rhoades  and  McGrath  (1996),  who  demonstrated  that  the  need  for 
technological  assistance  was  reduced  as  users  became  more  experienced  with  the  program.  In 
addition,  Olson  and  Olson  (2000)  demonstrated  that  students  work  more  effectively  as  they 
overcome  technological  problems.  The  initial  difficulties,  therefore,  may  be  considered 
“growing  pains,”  or  part  of  the  learning  curve  where  students  become  proficient  with  the 
particular  technology. 

Low  course  attrition  during  Phase  lb.  Attrition  is  a  pervasive  problem  in  DL  courses  in 
higher  education  and  training  environments  (Olson  &  Wisher,  2002;  Phipps  &  Merisotis,  1999). 
In  the  focus  group  discussions,  students  were  asked  if  they  had  ever  considered  dropping  out  and 
why.  Seven  of  the  eight  students  never  considered  it  because  they  needed  the  course  for  career 
progression.  Lack  of  time,  was  the  reason  given  by  the  one  student  who  contemplated  dropping 
out,  but  did  not.  The  most  common  reason  for  completing  the  course  was  that  the  course 
provided  opportunity  for  career  progression. 

Cohesiveness  in  the  learning  environment.  Students’  opinions  were  mixed  regarding  the 
effectiveness  of  the  team  building  process  during  the  synchronous  sessions.  One  group  reported 
that,  although  they  had  never  met  face-to-face  during  Phase  lb,  they  were  truly  a  team  before 
arriving  for  Phase  2.  These  students  said  that  text  messaging  allowed  for  both  training  task 
communication  and  social  interaction.  It  enabled  group  members  to  get  to  know  one  another’s 
strengths  and  weaknesses.  In  addition,  everyone  in  this  group  stated  that  allowing  different 
students  to  assume  various  leadership  roles  helped  them  learn  to  be  “followers”  and  trust  others. 
The  second  group,  however,  stated  that  they  did  not  feel  much  of  a  sense  of  group  cohesiveness, 
but  instead  were  like  “individuals  struggling  to  work  together  as  a  team.”  At  this  point,  there  is 
no  clear  reason  for  the  difference  in  opinions  between  the  groups,  and  the  question  of  what 
influences  group  cohesiveness  warrants  further  investigation. 

Instructor ’s  role  as  a  facilitator.  The  eight  students  agreed  that  the  role  of  the  instructor 
during  Phase  lb  was  more  of  a  facilitator  than  a  lecturer.  In  accordance  with  the  original  course 
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design  goals,  a  key  role  of  the  online  instructors  was  to  provide  direction  and  guidance  that 
facilitated  learning  instead  of  just  lecturing.  The  students  stated  that  feedback  from  the  instructor 
was  important,  and  therefore  they  sought  feedback  on  a  consistent  basis  throughout  the  course  in 
order  to  assess  their  own  progress. 

One  popular  pedagogical  technique  used  by  the  instructors  in  this  course  was  early  online 
introductions,  allowing  students  and  the  instructor  to  get  to  know  each  another.  The  purpose  of 
these  introductions  was  to  develop  common  ground  among  the  students  and  instructors.  Also,  as 
instructors  monitor  the  students’  text  messaging,  they  can  make  a  concerted  effort  to  get 
everyone  to  participate  and  to  contribute  to  the  real-time  discussions.  According  to  the  students, 
these  facilitative  techniques  used  by  the  instructors  were  successful  in  helping  the  students 
succeed  in  the  course. 

Perceived  advantages.  Since  the  course  was  designed  for  reservists,  most  of  whom  had 
full-time  jobs  and  families,  it  was  not  surprising  that  the  students  mentioned  convenience  as  the 
primary  advantage  for  the  online  course.  The  use  of  a  DL  format  allowed  the  students  to 
complete  the  course  according  to  their  own  schedules. 

Also  important  to  students  was  the  “active”  learning  environment  embedded  in  the 
course.  They  appreciated  the  immediate  feedback  and  mentoring  from  instructors,  and  most 
students  were  not  hesitant  to  contact  the  instructor  via  e-mail  for  such  feedback.  Some  students 
also  stated  that  they  learned  to  work  as  a  team  while  online,  and  that  the  online  environment 
fostered  more  thoughtful  comments  than  normally  found  in  conventional  classroom  settings. 

Perceived  disadvantages.  The  students  stated  that  a  key  disadvantage  of  the  course 
involved  the  length  of  the  asynchronous  and  synchronous  components.  This  is  in  agreement 
with  both  Sanders  and  Guyer  (2001)  and  Sanders  and  Burnside  (2001),  who  reported  that 
students  felt  the  asynchronous  portion  (Phase  la)  of  AC3-DL  was  too  long  and  filled  with  too 
much  detailed  information.  During  Phase  la,  the  tests  for  each  completed  section  used  different 
formats,  which  students  found  confusing.  In  addition,  when  students  did  complete  a  module, 
they  could  not  proceed  to  the  next  module  without  instructor  permission,  which  detracted  from 
the  “learn  anywhere,  learn  anytime”  benefit  of  online  learning.  This  situation  led  most  of  the 
students  to  skip  portions  of  Phase  la  just  to  get  through  it,  while  others  simply  dropped  the 
course1. 

In  contrast,  the  students  claimed  that  the  synchronous  portion  (Phase  lb)  was  too  short. 
Some  students,  for  instance,  mentioned  that  the  length  of  Phase  lb  did  not  allow  every  group 
member  to  play  all  roles  in  the  simulated  exercises.  They  suggested  that  additional  VTOC 
sessions  would  allow  all  members  to  experience  the  XO  (Executive  Officer)  and  S-3  (Operations 
and  Training)  positions. 


1  Initial  attrition  rates  for  the  first  phase  of  AC3-DL  was  50  percent  for  students  who  actively  participated  and 
approached  75  percent  if  all  students  who  enrolled  were  included  (Sanders  &  Burnside,  2001).  Discussions  with 
AC3-DL  course  developers  and  instructors  indicated  that  the  attrition  rate  was  significantly  reduced  in  later  cohorts 
as  the  course  delivery  methods  were  modified. 
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Students  were  concerned  with  the  reliability  of  the  VTOC  system  and  usefulness  of  some 
of  its  components.  At  times,  the  program  removed  students  from  the  messaging  rooms  without 
warning,  causing  the  loss  of  valuable  training  time  while  disrupting  workflow.  In  addition,  the 
map  editing  tool  tended  to  freeze  some  computer  systems,  and  the  terrain  viewer  was  not  found 
useful.  Most  students  stated  that  they  did  not  like  the  avatar  representing  their  physical  presence 
in  the  VTOC  operations  areas  and  that  the  VTOC  was  not  compatible  with  the  Macintosh 
computer  platform. 

To  alleviate  some  of  these  problems,  students  recommended  that  the  course  designers  and 
instructors  focus  on  basic  technology  shown  to  function  effectively  rather  than  the  latest  gadgets 
or  innovations.  This  is  not  surprising,  since  research  shows  that  online  instructors  and  students 
tend  to  rely  on  simple  tools  such  as  e-mail,  Web  links  to  course  material,  and  posting  lecture 
notes  online  (Bonk,  2001 ;  Bonk  2002).  Significantly  fewer  instructors  use  online  chatrooms, 
multimedia  lectures,  online  examinations,  animation,  and  streaming  video.  Using  new 
technology  (e.g.,  voice  over  the  Internet)  can  be  a  mistake  if  it  does  not  add  perceived  relevance 
and  effectiveness  to  the  course  or  solve  a  key  problem. 

In  addition  to  technological  concerns,  students  mentioned  a  few  other  disadvantages.  For 
instance,  the  online  examinations  during  Phase  la  offered  minimal  feedback  other  than  test 
scores.  They  also  stated  that  some  of  the  modules  could  have  been  delivered  and  tested  in 
smaller  chunks,  thereby  focusing  on  specific  accomplishments.  Students  seemed  concerned  with 
information  overload  and  a  reduced  sense  of  accomplishment  early  in  the  course,  which  could 
result  in  attrition. 

Interviews  with  Course  Instructors 

The  interviews  with  course  instructors  yielded  primarily  positive  results.  Topics  covered 
included:  a)  their  role  during  Phase  la  and  Phase  lb,  b)  the  instructional  techniques  they  used, 
and  c)  the  perceived  advantages  and  disadvantages  of  presenting  the  course  in  its  current  form. 
Interviews  with  the  course  instructors  provided  further  insight  into  the  pros  and  cons  of  teaching 
this  complex,  Web-based  course. 

Instructor ’s  role  as  facilitator.  The  online  instructors  did  not  view  their  roles  as  much 
different  than  in  the  classroom.  While  instructors  reported  that  they  served  more  of  a  facilitative 
role,  providing  students  with  the  means,  tools,  and  guidance  to  leam  effectively,  they  contended 
that  the  only  major  differences  between  teaching  online  and  teaching  in  a  classroom  were  that:  a) 
they  could  not  see  their  students  and  b)  instead  of  writing  grades  on  students’  assignments,  they 
sent  them  e-mails.  Two  instructors  emphasized  that  they  allocated  the  bulk  of  their  time  to 
course  planning,  but  the  third  instructor  indicated  that  he  had  spent  much  more  time  on  the 
administrative  aspects  of  the  course.  They  all  mentioned  the  importance  of  facilitating  good 
decision-making  and  problem  solving  so  that  students  can  apply  what  they  leam  to  real-life 
exercises. 

Instructional  techniques.  The  instructors  stated  that  particular  instructional  strategies  and 
pedagogical  approaches  were  useful  in  specific  online  environments.  During  the  asynchronous 
phase,  instructors  used  directive  and  one-way  instructional  techniques  aimed  at  teaching  basic 
concepts  and  information,  while  during  the  synchronous  phase  instructors  used  indirect 
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questioning,  prompting,  and  nudging.  It  is  possible  that  during  the  asynchronous  phase,  the 
students  may  have  needed  a  structured  approach  because  they  lacked  background  knowledge.  In 
contrast,  during  the  synchronous  phase  the  students  may  have  had  a  deeper  level  of 
understanding  to  build  from  and  were  able  to  respond  well  to  constructivist  techniques.  This 
difference  in  teaching  techniques  may  have  been  due  to  the  asynchronous-synchronous 
difference,  or  it  may  be  due  to  other  variables,  such  as  the  students’  level  of  understanding 
during  a  particular  phase. 

The  instructors  indicated  that  they  were  genuinely  interested  in  student  progress 
throughout  the  course  and  made  attempts  to  be  available  to  the  students.  For  example,  they  sent 
out  weekly  reminders  about  assignments  and  used  indirect  questioning  and  prompting  to  engage 
students.  Where  possible,  they  also  attempted  to  place  each  person  in  a  leadership  position  within 
their  groups  to  boost  individual  confidence  levels.  Instructional  tactics,  such  as  selecting 
students  to  be  in  charge  of  activities,  were  intended  to  boost  student  participation  during  the 
synchronous  component.  The  instructors  also  found  that  matching  weaker  students  with  strong 
leaders  was  beneficial.  In  fact,  they  noted  that  this  often  resulted  in  the  respective  groups 
supporting  poor  performers  on  their  own  with  little  help  needed  from  the  instructors. 

Perceived  advantages.  As  a  whole,  the  instructors  stated  that  the  online  course  fit  nicely 
into  a  small  group  instruction  model  and  strongly  complemented  the  Army’s  “crawl,  walk,  run” 
philosophy  of  learning.  In  effect,  the  asynchronous  portion  (Phase  la)  provided  the  basic 
foundation  (“crawl”),  the  synchronous  portion  (Phase  lb)  allowed  the  students  to  put  their 
knowledge  to  use  in  electronic  and  paper  formats  (“walk”),  and  the  resident  portion  (Phase  2) 
prompted  the  students  to  fully  apply  their  knowledge  and  skills  in  real-life  scenarios  (“run”). 

No  instructor  reported  difficulties  with  the  technology  or  the  instructional  methodology. 
One  of  the  advantages  of  the  Web-based  course  mentioned  was  the  ability  to  provide  detailed 
feedback  to  students,  which  promoted  greater  learning  and  application  of  knowledge  by  students. 
In  terms  of  the  asynchronous  phase,  the  instructors  claimed  that  students  were  effectively 
mentored  as  they  progressed  through  the  course  learning  new  concepts  and  ideas.  Moreover, 
they  felt  that  feedback  was  promptly  provided  during  Phase  la  and  was  based  on  progress  as 
well  as  performance.  Most  of  this  feedback  was  provided  through  e-mail.  Additionally,  the 
synchronous  component  (Phase  lb)  offered  students  immediate  feedback  from  both  instructors 
and  peers  using  either  messaging  or  voice  communication.  Overall,  the  course  instructors  gave 
the  course  design  a  positive  rating. 

The  instructors  stated  that  another  key  advantage  of  the  synchronous  course  sections 
involved  teaching  students  how  to  work  with  others  as  a  team  to  solve  a  problem.  In  addition  to 
problem  solving  and  teamwork,  instructors  contended  that  using  Web-based  communication 
applications  enhanced  students’  communication  skills.  The  rise  of  communications  technology 
and  the  need  for  team  skills  in  most  work  settings  support  claims  that  distance  technologies  have 
a  positive  impact  and  promote  important  skills. 

As  an  added  benefit,  instructors  mentioned  that  interactive  courses  could  give  students 
the  most  current  and  updated  material,  equipping  reservists  with  skills  and  training  equal  to  that 
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of  active  duty  soldiers.  The  DL  course  also  provides  an  additional  avenue  for  those  who  want  to 
advance  their  military  careers  and  earn  credits  towards  an  advanced  university  degree. 

Overall,  the  instructors  stated  that  they  enjoyed  teaching  the  course  online  and  using  the 
technology.  The  instructors  reported  that  the  students  going  through  the  DL  course  were  better 
trained  than  those  taking  it  through  a  previous  correspondence  version.  They  added  that  DL 
students  gain  general  skills  such  as  problem  solving  and  group  communication  that  are 
applicable  to  any  position  in  the  Army  and  could  not  be  gained  from  a  correspondence  course. 

In  fact,  they  recommended  replacing  all  correspondence  courses  with  distributed  learning 
versions,  especially  for  Army  recruiters  and  commanders  spread  out  across  the  United  States. 

Perceived  disadvantages.  While  these  instructors  did  not  indicate  many  disadvantages 
with  the  overall  course,  they  did  report  fairly  high  attrition  rates  during  the  asynchronous  phase 
(Phase  la).  They  sensed  that  part  of  the  problem  was  not  being  able  to  control  the  size  of 
modules.  As  a  result,  these  students  had  to  fit  a  fairly  robust  and  demanding  curriculum  into 
their  already  full  lives.  According  to  the  instructors,  early  modules  in  the  asynchronous  phase 
were  particularly  difficult.  However,  those  who  made  it  to  the  later  modules  usually  had  the 
stamina  and  motivation  to  complete  the  entire  course. 

Interview  with  DL  Education  Advisor 

Overall  perspective.  The  DL  Education  Advisor  for  the  Armor  School  provided  some 
valuable  information  concerning  the  design  of  DL  courses,  which  echoed  many  of  the  findings 
from  interviews  with  students  and  instructors.  The  DL  Education  Advisor  indicated  that  she  was 
quite  pleased  with  the  program  and  was  an  avid  supporter.  At  the  same  time,  she  was  interested 
in  additional  course  evaluation,  specifically  how  to  improve  student  completion  rates,  fine-tune 
course  production  and  system  resources,  and  enhance  online  Web-based  instruction  tools  and 
strategies.  She  emphasized  the  fact  that  anyone  involved  in  the  development  of  DL  technology 
must  be  flexible  and  adaptable.  The  technology  is  changing  so  rapidly  that  one  cannot  just  look 
at  where  the  technology  is  now,  but  must  also  consider  where  it  will  be  a  year  from  now. 

The  DL  Education  Advisor  said  that  applying  the  skills  learned  during  the  course  was  key 
to  its  success.  She  pointed  to  practical  exercises  that  were  embedded  in  the  course  to  help 
students  learn  the  content.  For  the  purpose  of  training  students  to  command  companies  and 
perform  other  duties  at  battalion  and  brigade  levels,  it  was  imperative  to  focus  on  bottom-line 
command  readiness. 

Advice.  For  those  wanting  to  replicate  aspects  of  this  program,  the  DL  Education 
Advisor  provided  several  caveats  and  tips.  She  offered  six  important  considerations  in  the 
design  of  DL  courses.  First,  all  courses  should  involve  direct,  e-mail  feedback.  Her  rationale 
was  that  students  need  to  feel  connected  to  other  students  and  to  the  instructor.  Second,  she 
claimed  that  courses  should  have  meaningful  content  that  students  can  directly  apply  to  real-life 
exercises.  Concerning  the  AC3-DL  course,  meaningful  content  typically  involved  combat 
situations  and  combat  readiness.  Third,  there  should  be  minimal  extraneous  content  (e.g.,  extra 
graphics  and  non-essential  information)  to  limit  student  confusion  and  course  complexity.  Her 
team  found  that  students  skipped  optional  or  peripheral  materials  if  they  felt  overwhelmed. 
Fourth,  designers  of  DL  courses  should  carefully  analyze  their  target  audiences  so  they  can 
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accurately  determine  what  the  students  want  and  need.  Fifth,  it  is  vital  to  create  an  active 
learning  environment  with  a  balance  between  flexibility  and  learner  accountability.  Not 
surprisingly,  she  readily  admitted  that  the  instructor  is  a  key  part  of  that  environment,  providing 
direction  and  feedback.  With  prompt  instructor  feedback,  students  were  not  isolated  in  their 
online  learning  endeavors.  Lastly,  designers  should  limit  their  visions  and  not  stretch  the 
expectations  of  technology  too  far  beyond  the  tools  and  options  that  have  been  proven  to  work. 
When  technology  is  pushed  to  the  limits  of  its  capabilities,  then  breakdowns  are  more  likely. 

The  DL  Education  Advisor  argued  that  there  would  always  be  room  for  improvement  in  the 
future,  but  that  one  has  to  start  somewhere. 

Limitations.  The  DL  Education  Advisor  noted  several  problems  with  the  current  system. 
First,  many  students  wanted  printed  copies  of  the  course  materials.  Given  the  online  availability 
of  the  course  materials,  however,  she  felt  that  this  would  amount  to  an  excessive  waste  of  paper. 
Second,  the  learner-management  system  was  not  flexible  enough  for  most  of  the  students  and 
instructors.  For  instance,  some  students  voiced  frustration  that  they  could  not  move  on  to 
another  volume  if  they  missed  too  many  items  on  the  gate  test.  Third,  since  there  was  not 
ubiquitous  access  to  the  Internet,  some  activities  and  events  were  not  always  available  to 
students  across  settings.  Fourth,  in  addition  to  Internet  access,  some  students  wanted  access  to 
course  materials  via  CD-ROM.  The  DL  Education  Advisor  noted  that  the  reason  for  Internet 
delivery  of  the  course  was  so  that  student’s  progress  through  the  course  could  be  easily  tracked. 

SUMMARY 

Previous  research  specific  to  this  course  found  that  the  blended  approach,  a  combination 
of  Web-based  and  face-to-face  lessons,  used  in  the  AC3-DL  course  was  at  least  as  effective  as 
the  correspondence  version  (Sanders  &  Burnside,  2001;  Sanders  &  Guyer,  2001).  In  the  initial 
version  of  the  AC3-DL  course,  the  synchronous  section  was  not  started  until  the  asynchronous 
phase  was  completed.  The  course,  however,  has  been  modified  once  again,  so  that  the 
asynchronous  and  synchronous  phases  are  intermingled.  This  approach  seems  to  be  time 
efficient,  allowing  students  to  complete  the  training  in  a  shorter  timeframe. 

Communication  Trends 

The  communication  trends  noted  during  synchronous  training  (Phase  lb)  provide  insight 
to  the  developmental  process  during  virtual  team  training.  During  the  initial  sessions,  time  and 
resources  must  be  allocated  for  social  bonds  to  develop  and  for  the  familiarization  of  technology. 
Course  developers  and  instructors  who  incorporate  computer-supported  collaborative  teams  into 
curricula  should  be  aware  of  the  time/resource  requirements  for  both  social  interaction  and 
developing  reasonable  skills  with  the  course  technology. 

In  the  current  research,  a  majority  of  the  messaging  discussions  focused  on  the  task. 
Although  always  high,  there  was  an  increased  likelihood  that  task-related  text-messaging 
instances  occurred  during  the  middle  sessions  rather  than  the  beginning  or  end.  This  was 
unanticipated,  because  the  exercises  across  all  meetings,  similar  in  content  and  difficultly, 
required  the  same  attention  on  task.  However,  the  findings  on  social  interaction  may  point  to  a 
reason  for  the  inverted  U  shape  found  in  the  task  data.  During  the  first  and  last  session,  there 
were  elevated  levels  of  social  text  messages.  The  text  messaging  data  showed  that  the  first 
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session  was  influenced  by  initial  greetings  and  getting  to  know  one  another  conversations. 

During  the  last  session,  individuals  spent  quite  a  bit  of  time  saying  goodbye  and  planning  to  meet 
up  and  carpool  to  the  “in-residence”  training  which  followed.  This  social  chat  trend  was 
complementary  to  the  task  related  messaging  trends;  when  social  messaging  decreased,  task 
related  messaging  increased. 

Participant  Reactions 

Qualitative  interviews  were  useful  in  identifying  aspects  of  the  course  that  made  it 
successful  or  inhibited  it  from  being  even  more  successful.  While  the  participants  stated  that 
length  of  Phase  la  was  a  problem  leading  to  high  levels  of  attrition,  they  also  acknowledged  that 
all  course  material  should  be  covered.  The  recommendation  of  the  participants  was  that  material 
should  be  segmented  into  more  sections,  so  that  they  can  be  learned  in  smaller  chunks. 

The  course  supervisor  and  the  instructors  offered  a  different  perspective  to  the  course. 

One  suggestion  was  that  the  development  of  a  DL  course  should  focus  on  content  and  not  on  the 
tools  of  technology.  This  is  in  agreement  with  prior  research  (Wisher  &  Cumow,  1999) 
demonstrating  that  additional  non-content  related  information  did  not  improve  learning. 

Interviews  with  students,  course  instructors,  and  the  course  supervisor,  revealed  some 
interesting  findings.  During  the  synchronous  DL  phase,  the  role  of  the  instructor  was  that  of  a 
facilitator  instead  of  a  lecturer.  Also,  learning  how  to  use  the  technology  was  a  concern  to  both 
the  students  and  the  instructors.  Therefore,  the  course  supervisor  suggested  that  for  instructional 
purposes  course  designers  should  not  attempt  to  stretch  the  technology  to  the  limits  of  its 
capabilities  because  that  increases  the  frequency  of  problems.  Taken  together,  the  results  from 
the  analysis  of  text  chat  communication  and  participant  reactions  provide  a  broad  picture  of  an 
effective  blended  approach  to  course  delivery. 
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APPENDIX  A  -  FOCUS  GROUP  QUESTIONS 


Questions  to  Students 

1 .  What  was  the  most  facilitative  aspect  of  the  online  chats? 

2.  What  were  the  advantages  of  taking  this  course  online  versus  in  a  classroom? 

3.  How  did  you  view  the  role  of  the  instructor  in  this  course?  Did  this  view  change  over  the 
course  of  the  class?  What  did  you  like  best  about  the  instructor?  What  did  you  like  least 
about  the  instructor? 

4.  How  did  the  tools  foster  student  interaction? 

(terrain  viewer,  map  edit,  text  chat,  voice  chat,  bookshelf,  word,  role-playing  as  Avatar, 
e-mail,  Spearhead  simulation) 

5.  What  advice  could  you  give  people  just  entering  this  course  concerning  the  most 
effective  ways  to  interact  with  others  online? 

6.  How  did  you  know  that  you  were  “off-track”  when  communicating  with  others  online? 

7.  Did  you  have  a  sense  that  someone  was  facilitating  or  leading  the  chars?  How  did  you 
know? 

8.  When  did  you  use  the  online  chats  versus  the  audio  chats? 

9.  What  were  the  most  common  technology-based  problems  that  you  encourtered 
throughout  the  course? 

10.  What  methods  did  you  use  to  develop  a  sense  of  “team  identity”  with  your  other  team 
members? 

1 1 .  How  often  did  you  seek  feedback  from  the  instructor?  How  often  did  you  seek  feedback 
from  peers?  What  methods  did  you  use  to  obtain  this  feedback? 

12.  What  percentage  of  time  during  the  course  did  you  work  without  interaction  with  another 
team  member  or  the  instructor? 

13.  Did  one  person  emerge  as  the  “leader”  of  the  team  during  the  course?  How  did  this 
individual  behave  to  indicate  to  you  that  he  was  the  leader? 

14.  How  did  you  and  the  other  group  members  resolve  conflicts  that  arose? 

15.  Do  you  see  the  impact  of  what  you  learned  in  Phase  I  training  in  Phase  II?  Do  any  of  the 
simulation  skills  transfer?  If  so,  how  do  they  help  you  in  Phase  II? 
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16.  How  often  do  you  play  simulation-based  computer  games?  Did  that  help  you  in  Phase  II 
training? 

17.  Did  you  ever  think  of  dropping  out?  If  so,  when  and  why? 
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APPENDIX  B  -  INSTRUCTOR  INTERVIEW  QUESTIONS 


1 .  What  tools  do  you  think  were  the  most  helpful  in  getting  students  to  learn  online? 

2.  What  instructional  strategies  or  pedagogical  approaches  were  particularly  useful?  How 
do  you  know  that  they  were  useful? 

3.  How  did  you  assess  student  learning?  How  would  you  change  your  assessment  of 
student  learning  in  the  future? 

4.  What  did  you  like  best  about  teaching  online? 

5.  What  do  you  like  least  about  teaching  online? 

6.  What  advice  would  you  give  other  instructors  who  were  considering  teaching  a  course 
online? 

7.  What  advice  would  you  give  potential  funders  of  online  projects? 

8.  What  goals  did  you  have  in  mind  when  teaching  this  class? 

9.  Where  and  when  do  you  think  the  most  learning  took  place? 

1 0.  Would  you  teach  this  way  again?  Why  or  why  not? 

1 1 .  What  aspects  of  the  online  course  do  you  think  students  had  the  most  problems  with? 
Were  you  able  to  deal  with  those  problems  effectively? 

12.  How  did  you  view  your  role  as  an  instructor?  Do  you  think  your  role  would  have  been 
different  if  you  had  taught  in  a  classroom  instead  of  online?  How  would  it  be  different? 

13.  How  often  did  students  request  feedback?  How  did  you  provide  this  feedback? 

14.  Did  you  have  any  students  that  were  not  as  actively  participating  as  the  rest?  What 
methods  did  you  use  to  increase  their  participation? 

15.  What  percentage  of  time  did  you  allocate  to  planning  and  preparation?  Online  teaching? 
Administration?  Interaction  with  students?  Interaction  with  content? 

16.  Who  typically  drops  out  of  the  live  and  online  courses?  Why? 

17.  What  are  the  key  outcomes  you  want  from  the  online  course? 
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